1. Field of the Invention
Embodiments of the present invention relate to multi-spectral imaging systems such as color still cameras, video cameras, scanners and microscopes and more specifically to imaging systems that use fewer sensor elements than previous techniques for comparable image quality.
2. Background Information
Images herein can be considered signals whose amplitude may represent some optical property such as intensity, color and polarization which may vary spatially but not significantly temporally during the relevant measurement period. In color imaging, light intensity typically is detected by photosensitive sensor elements or photosites. An image sensor is composed of a two dimensional regular tiling of these individual sensor elements. Color imaging systems need to sample the image in at least three basic colors to synthesize a color image. We use the term “basic colors” to refer to primary colors, secondary colors or any suitably selected set of colors. We exclude color difference signals, many of which are used in popular color spaces, from the definition of basic colors. Furthermore, all references to red, green and blue should be construed to apply to any set of basic colors.
Color sensing may be achieved by a variety of means such as, for example, (a) splitting the image into three copies, separately filtering each into the basic colors, and sensing each of them using separate image sensors, or (b) using a rotating filter disk to transmit images filtered in each of the basic colors onto the same image sensor.
However, a very popular design for capturing color images is to use a single sensor but overlay each pixel with a color filter. The resulting mosaic of color filters is known as a color filter array (“CFA”). This includes the color stripe design wherein the value of each output pixel is determined by three sensing elements, one for each basic color, usually arranged in horizontal, vertical or diagonal stripes. This CFA yields red, green and blue images of equal resolution or, if some other color space is used, all color components of equal bandwidth. The color stripe CFA is still used in high end cameras such as the Panavision Genesis Digital Camera. Information about the Panavision Genesis camera may be obtained from Panavision Imaging at One Technology Place, Homer, N.Y. 13077. Newer CFA designs by Bayer (see FIG. 4 and B. E. Bayer, “Color imaging array”, Jul. 20, 1976. U.S. Pat. No. 3,971,065) and others (see K. Hirakawa and P. J. Wolfe, “Spatio-spectral color filter array design for enhanced image fidelity” in Proc. of IEEE ICIP, pages II: 81-84, 2007 and L. Condat, “A New Class of Color Filter Arrays with Optimal Sensing Properties”) make different trade-offs between luminance and chrominance bandwidths as well as the crosstalk between them.
In the paper “Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain”, Proc. IS&T/SID 10th Color Imaging Conf, pages 331-336, 2002, D. Alleysson, S. Susstrunk, and J. Herault analyzed electro-magnetic filtering performed by CFAs as amplitude modulation of the color signals in the spatial domain (as used herein the terms “demosaic” and “demosaick” are to be construed as input image reconstruction procedures and the terms “demosaicer” and “demosaicker” as input image reconstruction algorithms). This led to frequency domain image reconstruction techniques that viewed the problem as that of demultiplexing the luminance and chrominance signals via demodulation and filtering. See E. Dubois, “Frequency-domain methods for demosaicking of bayer-sampled color images”, IEEE Signal Processing Letters, 12(12):847-850, 200 and N. Lian, L. Chang, and Y. P. Tan, “Improved color filter array demosaicking by accurate luminance estimation” in IEEE International Conference on Image Processing, 2005, ICIP 2005, volume 1, 2005.
The complementary problem of designing CFAs with good frequency domain properties was attacked by D. Alleysson, S. Susstrunk, and J. Herault, “Linear demosaicing inspired by the human visual system”, IEEE Transactions on Image Processing, 14(4):439-449, 2005 wherein the doubling of the number of blue photosites in the Bayer CFA at the expense of Green photosites was suggested. This was followed by techniques to design CFAs directly in the frequency domain by K. Hirakawa and P. J. Wolfe, “Spatio-spectral color filter array design for enhanced image fidelity” in Proc. of IEEE ICIP, pages II: 81-84, 2007 and optimized by L. Condat, “A New Class of Color Filter Arrays with Optimal Sensing Properties”. These techniques fix the pattern of each basic color to consist of a small set of spatial “carriers”—two dimensional sinusoids and their aliases with appropriate frequencies, phases and amplitudes—and sum over the three basic colors to arrive at the final pattern. This pattern is then overlaid on the sensor. When an image formed by the camera's lens is filtered by the CFA, it is modulated by each of the carrier frequencies. The overlap of the modulation products of the 3 primaries induces a color transform and leads to a multiplex of luminance and chrominance signals modulated at different frequencies. As long as there is limited cross-talk between the color components, and the color transform is invertible the original color image can be recovered.
It is important to note the role of the CFA in determining the noise figure of the camera. The sensitivity of each photosite should be approximately uniform to control sensor saturation. Furthermore, the color transform should have a numerically stable inverse and the transmission of light through the CFA should be maximized.
An important consideration in the choice of sensor color space so far has been the high frequency content of chrominance signals. Well chosen color transforms result in chrominance signals with low high frequency content. This allows the chrominance signals to be placed close to each other and to the luminance signal in the frequency domain without significant cross-talk. See Y. Hel-Or, “The canonical correlations of color images and their use for demosaicing” and K. Hirakawa and P. J. Wolfe, “Spatio-spectral color filter array design for enhanced image fidelity” in Proc. of IEEE ICIP, pages II: 81-84, 2007 and L. Condat, “A New Class of Color Filter Arrays with Optimal Sensing Properties”.
Unless otherwise specified, we shall use the term “resolution” to refer to linear resolution and assume it to be equal in all directions. We will also use the term “resolution”, thus defined, interchangeably with the term “bandwidth”.
An important factor influencing the close packing of color component signals is the geometry of their spectra. We use the term spectra to refer to the spatial Fourier transform of the image. The maximum spatial frequency along any direction that can be captured by a sampling lattice is inversely proportional to the lattice pitch along that direction as per the Nyquist theorem. Square and rectangular sampling lattices admit higher bandwidth along the diagonal directions than along horizontal or vertical. Optical systems, on the other hand, generate roughly equal bandwidth along all directions thereby yielding images with nearly circular spectral support. This leads to the problem of efficiently packing circles into squares or rectangles.
An aggressive technique for close packing of color component spectra employs adaptive directional techniques during demosaicking. These techniques assume the color component spectra of small image patches to be sparse in at least one direction. They design their CFA to generate more than one copy of chrominance spectrum (see B. E. Bayer, “Color imaging array”, Jul. 20, 1976, U.S. Pat. No. 3,971,065), identify the cleanest copy during the demosaicking step and use directional filtering to demultiplex them (see Ron Kimmel, “Demosaicing: Image reconstruction from color ccd samples”, IEEE Trans. Image Processing, 8:1221-1228, 1999 and E. Chang, S. Cheung, and D. Y. Pan, “Color filter array recovery using a threshold-based variable number of gradients”, in Proceedings of SPIE, volume 3650, page 36, 1999 and K. Hirakawa and T W Parks, “Adaptive homogeneity-directed demosaicing algorithm”, IEEE Transactions on Image Processing, 14(3):360-369, 2005 and E. Dubois, “Frequency-domain methods for demosaicking of bayer-sampled color images”, IEEE Signal Processing Letters, 12(12):847-850, 2005). The benefits of adaptive directional demosaicking come at a heavy cost, though, since sensing edge directions from noisy sub-sampled images is a hard problem and the non-linear nature of decision making makes noise reduction a non-separable step.
For a frequency domain analysis of the popular Bayer CFA, see E. Dubois, “Frequency-domain methods for de-mosaicking of bayer-sampled color images”, IEEE Signal Processing Letters, 12(12):847-850, 2005. FIG. 4 shows the Bayer CFA 410. 710 in FIG. 7 illustrates how color information with its circular support is packed into the sensor's rectangular support. This can be most easily understood in terms of an alternative color space:
                              [                                                    L                                                                                      C                  ⁢                                                                          ⁢                  1                                                                                                      C                  ⁢                                                                          ⁢                  2                                                              ]                =                                            1              4                        ⁡                          [                                                                    1                                                        2                                                        1                                                                                                              -                      1                                                                            2                                                                              -                      1                                                                                                                                  -                      1                                                                            0                                                        1                                                              ]                                ⁡                      [                                                            R                                                                              G                                                                              B                                                      ]                                              (        1        )            
The central circle represents the Luminance (L) spectrum. The four quarter circles at the vertices make up Chrominance1 (C1). The two semi-circles at the left and right edges make up the first copy of Chrominance2 (C2a). The two semi-circles at the top and bottom edges make up the second copy of Chrominance2 (C2b).
It's apparent from this figure that there is empty space between circles that corresponds to unused frequencies. The present invention minimizes such inefficiency.
Existing sensor designs predominantly use square pixels. This includes sensors used with the Bayer CFA. Designs using rectangular sensor elements do exist. The grass valley Viper camera (see http://www.grassvalley.com/docs/Brochures/cameras/viper/viper_br.pdf, page 7) uses rectangular sensor elements so as to facilitate combining of multiple elements into single effective elements thereby realizing configurable sensor resolutions. For more information, Grass Valley can be reached at 400 Providence Mine Road, Nevada City, Calif. 95959.
Video systems that must maintain compatibility with analog NTSC or PAL broadcast television standards also use rectangular pixels. Scan rates of horizontal scanning lines in these analog systems control the vertical resolution and the video circuit bandwidth controls the horizontal resolution. The ITU (see ITU-R, “Recommendation BT.601-6: Studio encoding parameters of digital television for standard 4:3 and wide screen 16:9 aspect ratios”) specifies different pixel height and width to capture digital video with these different vertical and horizontal resolutions.
The color stripe sensor uses rectangular pixels wherein the longer sides are 3 times the length of the shorter. As it uses three primary colors, this gives it equal resolution in the vertical and horizontal directions.